Skip to content

[DRAFT] Vllm fused moe#3687

Draft
NuojCheng wants to merge 4 commits intomainfrom
chengnuojin-vllm-rule
Draft

[DRAFT] Vllm fused moe#3687
NuojCheng wants to merge 4 commits intomainfrom
chengnuojin-vllm-rule

Conversation

@NuojCheng
Copy link
Copy Markdown
Collaborator

@NuojCheng NuojCheng commented Apr 16, 2026 •
edited
Loading

Description

Example command

NEW_MODEL_DESIGN=1 python3 -m maxtext.inference.vllm_decode src/maxtext/configs/base.yml     model_name=qwen3-30b-a3b     tokenizer_path=Qwen/Qwen3-30B-A3B     vllm_hf_overrides='{architectures: ["MaxTextForCausalLM"]}' load_parameters_path=gs://parambole-qwen3-moe-verification/unscanned/qwen3-30b-a3b-thinking-2507/14_08_2025/0/items     ici_expert_parallelism=4     hbm_utilization_vllm=0.5     prompt="Tell me three fun facts about Buenos Aires."     decode_sampling_temperature=0.0     decode_sampling_nucleus_p=1.0     decode_sampling_top_k=0.0     pure_nnx_decoder=True     use_chat_template=True     sparse_matmul=True     prefuse_moe_weights=True     debug_sharding=True 2>&1 | tee  qwen3_30b_vllm_ep.log

Tests

Please describe how you tested this change, and include any instructions and/or
commands to reproduce.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@NuojCheng NuojCheng force-pushed the chengnuojin-vllm-rule branch from bda614a to 4d486c4 Compare April 17, 2026 17:49
@NuojCheng NuojCheng added the draft Draft PR label Apr 17, 2026
@NuojCheng NuojCheng force-pushed the chengnuojin-vllm-rule branch from 8154d78 to aa840a7 Compare April 17, 2026 21:55
@NuojCheng NuojCheng force-pushed the chengnuojin-vllm-rule branch from aa840a7 to 0f4beb8 Compare April 17, 2026 23:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vipannalla vipannalla Awaiting requested review from vipannalla vipannalla will be requested when the pull request is marked ready for review vipannalla is a code owner

@mitalisi mitalisi Awaiting requested review from mitalisi mitalisi will be requested when the pull request is marked ready for review mitalisi is a code owner

@gpolovets1 gpolovets1 Awaiting requested review from gpolovets1 gpolovets1 will be requested when the pull request is marked ready for review gpolovets1 is a code owner

@mailvijayasingh mailvijayasingh Awaiting requested review from mailvijayasingh mailvijayasingh will be requested when the pull request is marked ready for review mailvijayasingh is a code owner

@jrplatin jrplatin Awaiting requested review from jrplatin jrplatin will be requested when the pull request is marked ready for review jrplatin is a code owner

@patemotter patemotter Awaiting requested review from patemotter patemotter will be requested when the pull request is marked ready for review patemotter is a code owner

@Lumosis Lumosis Awaiting requested review from Lumosis Lumosis will be requested when the pull request is marked ready for review Lumosis is a code owner

@richjames0 richjames0 Awaiting requested review from richjames0 richjames0 will be requested when the pull request is marked ready for review richjames0 is a code owner

@RissyRan RissyRan Awaiting requested review from RissyRan RissyRan will be requested when the pull request is marked ready for review RissyRan is a code owner

@michelle-yooh michelle-yooh Awaiting requested review from michelle-yooh michelle-yooh will be requested when the pull request is marked ready for review michelle-yooh is a code owner

@gagika gagika Awaiting requested review from gagika gagika will be requested when the pull request is marked ready for review gagika is a code owner

@shralex shralex Awaiting requested review from shralex shralex will be requested when the pull request is marked ready for review shralex is a code owner

@suexu1025 suexu1025 Awaiting requested review from suexu1025 suexu1025 will be requested when the pull request is marked ready for review suexu1025 is a code owner

@jesselu-google jesselu-google Awaiting requested review from jesselu-google jesselu-google will be requested when the pull request is marked ready for review jesselu-google is a code owner

@parambole parambole Awaiting requested review from parambole parambole will be requested when the pull request is marked ready for review parambole is a code owner

@shuningjin shuningjin Awaiting requested review from shuningjin shuningjin will be requested when the pull request is marked ready for review shuningjin is a code owner

@jiangjy1982 jiangjy1982 Awaiting requested review from jiangjy1982 jiangjy1982 will be requested when the pull request is marked ready for review jiangjy1982 is a code owner

@gobbleturk gobbleturk Awaiting requested review from gobbleturk gobbleturk will be requested when the pull request is marked ready for review gobbleturk is a code owner

@bvandermoon bvandermoon Awaiting requested review from bvandermoon bvandermoon will be requested when the pull request is marked ready for review bvandermoon is a code owner

@NicoGrande NicoGrande Awaiting requested review from NicoGrande NicoGrande will be requested when the pull request is marked ready for review NicoGrande is a code owner

@abhinavclemson abhinavclemson Awaiting requested review from abhinavclemson abhinavclemson will be requested when the pull request is marked ready for review abhinavclemson is a code owner

@khatwanimohit khatwanimohit Awaiting requested review from khatwanimohit khatwanimohit will be requested when the pull request is marked ready for review khatwanimohit is a code owner

@SurbhiJainUSC SurbhiJainUSC Awaiting requested review from SurbhiJainUSC SurbhiJainUSC will be requested when the pull request is marked ready for review SurbhiJainUSC is a code owner

@hengtaoguo hengtaoguo Awaiting requested review from hengtaoguo hengtaoguo will be requested when the pull request is marked ready for review hengtaoguo is a code owner

@A9isha A9isha Awaiting requested review from A9isha A9isha will be requested when the pull request is marked ready for review A9isha is a code owner

@aireenmei aireenmei Awaiting requested review from aireenmei aireenmei will be requested when the pull request is marked ready for review aireenmei is a code owner

@dipannita08 dipannita08 Awaiting requested review from dipannita08 dipannita08 will be requested when the pull request is marked ready for review dipannita08 is a code owner

@igorts-git igorts-git Awaiting requested review from igorts-git igorts-git will be requested when the pull request is marked ready for review igorts-git is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

draft Draft PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@NuojCheng @NicoGrande